(Measurement) Errors in Academic
Publications: Statistical Discussion
Tim Houle, PhD
Massachusetts General Hospital
Conflicts of Interest
• GlaxoSmithKline
– Statistical Consultant
• StatReviewer
– Chief Scientist
• Anesthesiology
– Statistical Editor
Overview
• Statistical/Methodological Errors in
Observational Research
– Most recent
• Measurement Error In Statistical Models
– 5 myths about error
van Smeden M, Lash TL, Groenwold RH, van Smeden M. Five myths about measurement error
in epidemiologic research.
Common Errors in Academic
Publications: Reporting
• Propensity
– Model specification
– Model diagnostics
• Calibration
• Overlap
• Matching
– Methods (probability, exact, etc.)
– Algorithm (random, greedy)
– Software
• Model
– Specification
– Distribution, Link
– Interaction
– Calibration
Common Errors in Academic
Publications
• Typographical errors or copy-paste errors
– “OR 1.75, 95%(CI: 0.45 to 0.95)”
• Data-driven confounder selection
– Stepwise variable selection
• Underdeveloped multiple imputation models
– “We used MI to replace missing data”
Measurement Error
• Mismeasurements and misclassifications of all
kinds
– Mistaken entries
– Inaccurate recordings
– Imperfectly reliable measurements
• Observed = True + Error
Types of Measurement Error
• Classical error
– ~ N(0,s2)
• Systematic error
– ~N(bias, s2)
• Differential
– Error dependent on outcome
• Berkson
– True + ~N(0, constant)
Measurement error is often neglected
in medical literature: a systematic
review
• Original research published in 2016 in high-
impact medical and epidemiology journals
– Main exposure or confounder
• Search strings related to “measurement error”
Brakenhoff TB, et al. Measurement error is often neglected in medical literature: a systematic
review. Journal of clinical epidemiology. 2018 Jun 1;98:89-97.
Measurement error is often neglected in
medical literature: a systematic review
• 1178 articles found, 565 met inclusion criteria
– 337 Epidemiology
– 228 High Impact Medical
• 247/565 (44%) directly addressed
measurement error
– 70% ONLY in the Discussion section
Brakenhoff TB, et al. Measurement error is often neglected in medical literature: a systematic
review. Journal of clinical epidemiology. 2018 Jun 1;98:89-97.
Myth 1:
Measurement error can be compensated by large number of
observations
• Increased N causes estimates to approach the
measurement error mechanism, not their true value
• With unreliable measurements, sample size needs to
increase ~ 50 fold to compensate (Devine, 1998)
• “Triple Whammy” (Carroll, 2006)
– Covariate-outcome relationship biased
– Statistical power diminished
– Relational features masked
• Non-linearity difficult to detect
Carroll et al.. Measurement Error in Nonlinear Models: A Modern Perspective. Chapmann & Hall/CRC; 2006.
Devine et al. Estimating sample size for epidemiologic studies: the impact of ignoring exposure measurement uncertainty.
Stat Med 1998;17:1375-1389.
Myth 2:
The exposure effect is underestimated when variables are
measured with error
• An exposure can be over or under estimated in
the presence of measurement error
• Spearman error attenuation formula
– Regression dilution bias
– Attenuation to the null
– Hausman’s iron law
Spearman C. The proof and measurement of association between two things. Am J Psychol
1904;15:72-101.
yyxx
xy
yx
rr
r
r
*
''
yx
xy
xy
liabilityliability
nAssociatio
Observed
nassociatioTrue
Re*Re
""
Observed Associations
SBP exposure and DBP confounder
r = 0.65
SBP exposure and ABI confounder
r = -0.17
Myth 3:
Exposure measurement error is nondifferentiable if
measurements are made without knowledge of outcome
Exposure
Outcome
Example: Case-control study where cases attend more to the existence of
an exposure
Myth 4:
Measurement error can be prevented but not mitigated in
observational data analysis
• Statistical methods for error bias adjustments
are available
– Knowledge of error structure
– Knowledge of error variance
• Greatly facilitated by a validation sample
– Observed data can be contrasted with ‘true’ data
– Repeated measures
– Surrogate measures (latent dimensions)
Measurement Error Correction
Methods
• Regression calibration (Rosner et al., 1989)
– Error prone covariate replaced by expected true score
• Simulation-extrapolation (SIMEX; Cook et al., 1994)
– Simulate dataset through adding MORE error to the covariates
– Extrapolate predictions back to original situation
• Latent variable models
– Replicate measures used to estimate ‘true’ value of a more reliable
latent construct
• Bayesian approaches
• Multiple Imputation
Myth 5:
Certain types of observational research are unaffected by
measurement error
• Single exposure and set of confounders
• Time-series analysis
• Diagnostic accuracy studies
• Randomized controlled trials
• Many others…
Recommendations
• Measurement error is nearly ubiquitous in observational data
analysis
– Let’s stop neglecting it
• Measurement error can have a counter-intuitive impact on
observed associations
– Must consider its structure AND degree
– Be cautious when applying general claims about the direction of the
error
• Strongly consider the use of formal strategies to mitigate error
– Conduct validation efforts
– Utilize formal statistical methods
van Smeden M, Lash TL, Groenwold RH, van Smeden M. Five myths about measurement error
in epidemiologic research.
Thank you!
• thoule1@mgh.harvard.edu
